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High-quality, well-implemented early childhood education (ECE) positively 


affects the learning trajectories of children who start school with lower 
skills than their peers, according to decades of evidence. Yet studies on 
ECE programs across the country reveal that too few offer high-quality 
programming. To date, the ECE field has focused most improvement 
efforts on classroom materials and interactions. Broadening these efforts 
to an organization-wide focus can better support quality improvement. 
The University of Chicago Consortium on School Research (UChicago 
Consortium) and the Ounce of Prevention Fund (Ounce) designed teacher 
and parent surveys, the “Early Education Essential Organizational 
Supports Measurement System” (Early Ed Essentials), to help ECE sites 
diagnose organizational strengths and weaknesses. 

The current study tested whether the newly-adapted and designed 
Early Ed Essentials teacher and parent surveys capture reliable and valid 
information about the organization of ECE programs—information that 
is also associated with existing indicators of program quality. 


Data Used in This Study: 


Quantitative Data Qualitative Data 

¢ Sample: 81 sites in Chicago ¢ Sample: 4 sites (from the quantitative sample) 
¢ 41 school-based and 40 community-based * 2 sites with high & 2 sites with low Early Ed 

* Data: Early Ed Essentials surveys, collected Essentials survey scores; 1 school- and 
winter-spring 2016: 1 community-based site for each 
¢ 745 teacher surveys ¢ Data: Site-visits lasting 3 consecutive days in 
° 2,464 parent surveys May-June 2016, using protocols to capture 

* Outcomes Used: Observational measures in-depth information about what the essential 
of teacher-student interactions (using the supports look like and how they are experi- 
CLASS Pre-K) and student attendance enced by staff and family members in ECE site 


¢ Data collection included: 
¢ Individual interviews of leaders and teachers, 
and group interviews of family members; 
* Observations of common area environments, 
activities, and interactions; 
« Photographic documentation of common 
area spaces and displays. 
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Research Findings 


Essential Supports Framework 
The Early Ed Essentials tool was created using the five essential supports framework’ and 
existing K-12 5Essentia/s surveys.? Researchers adapted the teacher survey and created a new 
parent survey for ECE settings, then tested for reliability and validity. 


Snapshot Figure 1 


Early Ed Essentials and 5Essentials Surveys are Comprised of Essentials, Measures, and Items 


bo te) af NAN ees) 
* Comprised of multiple measures 


MEASURES* 


ITEMS 


¢ Higher-order categories of organizational supports that are essential 


for improving student learning 


* Capture survey respondents’ beliefs or experiences of a single construct 


pees eae * Comprised of multiple items 


* Researchers used Rasch models to create the measures from sets of items; see http./bit.ly/RaschOverview for details. 


¢ Individual survey questions that capture a range of beliefs and experiences 


The Measures Are Grouped into 6 Essentials for This Study 


¢« In many ways, these align with the placement of measures into essentials on the K-12 5Essentia/s survey. 


* However, parents’ responses to the survey indicated different perspectives from teachers’ responses. 


¢ Therefore, we created a sixth essential to test in our validation analyses: parent voice. 


SNAPSHOT FIGURE 2 


Measures Included in the Early Ed Essentials 


ESSENTIALS 
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MEASURES 


Instructional Leadership* 


* Program Coherence 


Teacher Influence 
Teacher-Leader Trust 


Collective Responsibility* 
Teacher Collaboration 

School Commmitment 
Teacher-Teacher Trust 
Socialization of New Teachers 


Teacher Safety* 
Child-Child Interactions* 
Positive Learning Climate* 


ESSENTIALS 
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MEASURES 


Early Cognitive Development* 

Early Language Development* 

Early Math Development* 

Early Social-Emotional Development* 
Quality of Student Interactions* 


Teacher-Parent Trust® 
Parent Involvement 
Teacher Outreach and 
Collaboration with Parents* 
Parent Influence 


Including Parents as Partners*P 
Teacher Communication with Parents*? 
Program Orientation towards Early 
Education*? 

Parent Influence on the Program*? 


Note: * New ECE measure (not on K-12). & Adapted slightly from K-12 measure. P Parent survey measure. All other measures are on the teacher survey. 


1 Bryk, Sebring, Allensworth, Luppescu, & Easton (2010). 2 https://uchicagoimpact.org/tools-training/S5essentials 
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If the Early Ed Essentials surveys do measure organizational constructs that research suggests 
are important for ECE programs, then survey results should be positively related to established 
measures of ECE quality. On the other hand, we do not expect the survey data to map perfectly 
onto these other measures—the surveys should be providing consistent information while 
also identifying practices and experiences that other tools do not yet capture. 


« Most, but not all, essential scores were significantly related to site-level outcomes. 


¢ Neither ambitious instruction nor parent voice scores were significantly related to either outcome measured. 


SNAPSHOT FIGURE 3 SNAPSHOT FIGURE 4 
Essential Scores Related to CLASS Pre-K Essential Scores Related to Student Attendance 
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Note: Each pair of bars compares average CLASS scores/attendance rates between sites with essential scores in the bottom vs. top quartiles. Each site’s average CLASS score/ 
attendance rate was obtained by fitting unconditional 2-level HLMs with either classrooms (for CLASS) or students (for attendance) nested within sites; these model-fitted scores 
were then used to produce the top/bottom quartile average score. * indicates that the relationship between the essential score and the outcome is statistically significant at the 
p<0.05 level; ** indicates significance at the p<0.01 level; *** indicates significance at the p<0.001 level. 


€) mM Reliability 


It is critical to ensure that the measures on the Early Ed Essentia/s capture the true response 
(i.e., beliefs or experiences) of the respondent and do not have large amounts of random error. 
This tells us they are measuring what they’re intended to measure. Analyses showed: 


¢ All the measures on the surveys were reliable; supports. Measures that were less sensitive fell 
we can trust they are accurately measuring under the ambitious instruction and parent voice 
people’s beliefs and experiences on the construct essentials. 
being asked about. * The surveys did not have strong bias toward either : 
- Many measures were sensitive enough to detect ¢ School- or community-based ECE sites; or 
differences between sites; the surveys are well- ¢ English or Spanish speakers (parent survey only). 
designed to effectively capture site-wide beliefs ¢« This means the surveys can be used and compared 


or experiences with organizational essential across different types of sites. 


@Q hi “Practical” Validation: Observations and Interviews 


To provide additional evidence of discriminant validity and “practical” validation that the Early Ed 
Essentials are able to differentiate across sites, researchers asked: Are there qualitatively different 
climates, structures, and practices between ECE sites with high vs. low Early Ed Essentials survey 


scores? 


Interview and observation evidence confirmed that the Early Ed Essentials differentiates between ECE programs: 


¢ Staff and families in sites with high and low survey responses provided qualitatively different descriptions 
and experiences of organizational climate and conditions, summarized in Table 1. 


SNAPSHOT TABLE 1 
Reports of Organizational Climate and Conditions 


At ECE Sites with High Essential Support Scores: 


1. Staff held common understandings of their goals as an 
ECE program that were guided by their leader’s strong, 
purpose-driven vision rooted in child developmental 
science and developmentally-differentiated practice. 


2. Leaders built emotionally-encouraging relationships 
with staff, set up structures that protected time for cross- 
classroom collaboration, and used these relationships and 
routine discussions of practice to build a unity of purpose. 


3. Leaders, teachers, and staff emphasized the importance 
of children’s social-emotional learning as the foundation 
for all learning. All staff worked diligently to create 
supportive learning environments for children and their 
families. 


4. Leaders, teachers, and staff believed that partnerships 
with families were critical for effective teaching and 
children’s success, including the input of families on 
high-level instructional decisions. 


5. Interactions and conversations among staff, and between 
staff and parents, were frequent, warm, and focused on 
offering one another encouragement around endeavors— 
both professional and personal. 


IN THE WORDS OF A TEACHER: 


“1 feel like it’s empowering [here]...it’s not just 
from the top down. We believe in this stuff and | 
have something to share and it’s valued by your 
FXelaalialisiaa- ike) emu al-laPmYLel0] axeloban-¥-[elal-1acm-) ale m\Zol0l @ 
colleagues also buy in, too, and you have that 


energy and you have that love and then you have 
an administrator that pushes you in that way 
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you a bit farther.” 


— Teacher A 


At ECE Sites with Low Essential Support Scores: 


Staff articulated that their primary aim was making sure 
they complied with the myriad of program regulations 
consuming the focus of their leaders, including that 
children achieve program-established kindergarten- 
readiness goals. 


. There was an absence of leadership practices and 


organizational structures that advanced a pedagogical 
vision, coherently guided instruction, or allowed staff time 
to focus together on the work of teaching and learning. 


Leaders, teachers, and staff emphasized rote learning as 
the key strategy for preparing children for kindergarten. 
Staff described how children’s lack of self-regulation was 
a barrier to their teaching and children’s learning. 


. Leaders and teachers believed that family involvement 


in the program was not critical to teaching and learning. 
A subset of teachers believed families caused children’s 
difficulties adjusting to the classroom. 


. Teachers and staff kept to individual classrooms, inter- 


acting minimally with one another and families in the 
common areas and through brief, perfunctory exchanges. 


IN THE WORDS OF A TEACHER: 


“That’s one thing that gets to me [here]: 

There is no collaboration. | am used to going 

to another classroom and saying, ‘Hey, | couldn’t 
do it this way. Can you tell me how | can do it 
that way, or didn’t that way work for you?’ [But 
here] everybody is not even on the same plan. 
Everyone is not even using the same curriculum 
[name]. That’s what gets me because if | have a 
problem...and | want to compare, | can’t.” 


— Teacher B 


Implications 


fter several years of development and testing, the Ear/y Ed Essentials now has evidence that 
it provides valid and reliable data to educators, families, policymakers, and researchers. In 
the near future, the valid portions of these surveys will be available for use in early education 
settings. Other areas will continue to be under development. There is considerable promise 
for this measurement system in the years ahead. Key considerations for practice, policy, and 


research include: 


¢ Expanding the definition of “quality.” The Early 
Ed Essentials and the theoretical framework 
underlying the tool can help broaden the defini- 
tion of “quality” in ECE to include organizational 
conditions and the important role of leaders as 
instructional guides. This work underscores the 
critical role leaders play in shaping the quality of 
teaching through professional collaboration and 
community. These changes to the definition of 
quality could influence how programs are incen- 
tivized and resourced to galvanize improvement. 


Actionable data for improvement. Survey 
results can provide ECE programs with action- 
able data that leaders can use to focus attention 
on strengthening the organizational supports for 
teaching, learning, and family engagement. The 
Ounce is also producing tools to guide leaders in 
using the survey data to identify what steps they 
can take collaboratively with staff to improve 
weak areas. 


¢ ECE-K-12 alignment. The Ear/y Ed Essentials pro- 
vides language about ECE program quality that 
aligns with language that practitioners in elemen- 
tary schools use to discuss quality improvement. 
The alignment between the Early Ed Essentials 
and K-12 5Essentia/s tools provides a common 
lens, language, and metric for understanding 
and promoting instructional improvement across 
the educational continuum. Leaders from both 
sectors can strengthen alignment of structures 
and practices, and thus the experiences children 
and families have as they transition from pre- 
kindergarten into early elementary grades. 


¢« Research potential. The Early Ed Essentials has 
the potential to grow new areas of ECE research 
that can expand our understanding of program 
effectiveness, and how organizational climate and 


conditions are related to characteristics of ECE 
settings, staff, leaders, students, families, and/or 
communities. With few ECE measures currently 
available to understand organizational supports, 
these surveys provide a cost-effective way to 
gain staff and parent perspectives on the orga- 
nization’s climate and conditions. The Early Ed 
Essentials can also provide contextual information 
about programs that may shed light into other 
long-standing ECE research questions, such as 
why some programs seem to thrive and others do 
not, even when similar structural conditions exist. 


Limitations in the field. Because this study is 
the first of its kind on these surveys, there will 
be more to study and learn over time. When 
considering use of the Early Ed Essentials in 
the field, it’s important to recognize that: 


¢ The surveys and outcomes (CLASS and atten- 
dance) are related to each other, but we do not 
yet know whether essential supports in early 
education settings cause changes in outcomes. 

e Because these surveys were tested in ECE 
settings with education-focused classrooms 
serving 3- to 5-year-olds, we cannot assume 
they will work similarly in other settings, such 
as in infant/toddler classrooms. 

« Some areas of the surveys will continue to be re- 
fined and tested, including ambitious instruction 
on the teacher survey and the full parent survey. 

¢ We are in the early stages of developing mea- 
sures and researching the relationships between 
organizational conditions and outcomes within 
ECE programs. The authors therefore caution 
against using the Early Ed Essentia/s surveys 
as an accountability metric itself—at least until 
there is ample opportunity for the field to 
understand its use as an improvement tool. 
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ABOUT THE RESEARCH 


Information and evidence in this snapshot are derived 
from research by: 


Stacy B. Ehrlich, NORC at the University of Chicago 
(formerly the University of Chicago Consortium on 

School Research) 

Debra M. Pacchiano, Amanda G. Stein, & Maureen Wagner, 
The Ounce of Prevention Fund 


ABOUT THE UCHICAGO CONSORTIUM 


The University of Chicago Consortium on School Research 
conducts research of high technical quality that can inform 
and assess policy and practice in the Chicago Public Schools. 
We seek to expand communication among researchers, policy- 
makers, and practitioners as we support the search for solutions 
to the problems of school reform. The UChicago Consortium 
encourages the use of research in policy action and improve- 
ment of practice, but does not argue for particular policies or 
programs. Rather, we help to build capacity for school reform 
by identifying what matters for student success and school 
improvement, creating critical indicators to chart progress, and 
conducting theory-driven evaluation to identify how programs 
and policies are working. The UChicago Consortium is a unit of 
the Urban Education Institute. 
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Ehrlich, S.B., Pacchiano, D.M., Stein, A.G., Wagner, M.R., 
Luppescu, S., Park, S., Frank, E., Lewandowski, H., & 

Young, C. (2018). 

Organizing early childhood education for improvement: Testing 
anew survey tool. Chicago, IL: University of Chicago Consortium 
on School Research and the Ounce of Prevention Fund. 


ABOUT THE OUNCE OF PREVENTION FUND 


The Ounce of Prevention Fund 


INSPIRE MINDS. 
CHANGE LIVES.” 


is dedicated to giving all children, 
especially those from low-income 
communities and families, the best 
chance for success in school and in life. We provide, research, 
and advocate for high-quality early childhood education from 
birth to age 5, giving young children the physical, cognitive, 
social and emotional development they need to develop and 
learn. As a public-private partnership working at the intersec- 
tion of practice, policy, and research, the Ounce aims to be the 
nation’s most trusted resource for early childhood knowledge. 


Views expressed in this document are those of the authors and do not necessarily reflect those of the UChicago Consortium, the 


University of Chicago, or the Ounce of Prevention Fund. 
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